Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Arducam OV64A40 64MPx camera module #84

Closed
wants to merge 22 commits into from

Conversation

jmondi
Copy link
Contributor

@jmondi jmondi commented Nov 10, 2023

No description provided.

naushir and others added 19 commits October 18, 2023 14:58
Add a new subpoject wrap file for the libpisp library located at
https://github.com/raspberrypi/libpisp

The libpisp library is used to configure the Raspberry Pi 5 Frontend
and Backend ISP components.

Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Add the Raspberry Pi 5 PiSP specific compressed Bayer format types 1/2:
- V4L2_PIX_FMT_PISP_COMP1_xxx
- V4L2_PIX_FMT_PISP_COMP2_xxx

Add the Raspberry Pi 5 PiSP Frontend and Backend config formats:
- V4L2_META_FMT_RPI_FE_CFG
- V4L2_META_FMT_RPI_BE_CFG

Add the Raspberry Pi 5 PiSP Frontend statistics format:
- V4L2_META_FMT_RPI_FE_STATS

Add 16-bit Bayer formats:
- MEDIA_BUS_FMT_Sxxx16_1X16

Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
We can perform some of the local contrast adjustment using global
gains in the LSC table. We can vary the amount of gain according to
the measured brightness of that image region.

Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
This was being re-read in order to determine what LSC gains had been
applied. We can just retrieve these numbers from the prevAsyncResults_
instead.

This will also enable other future algorithms to manipulate the LSC
tables in the alsc.status, without it breaking the core ALSC algorithm
here.

Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
"Fast desaturation" is a technique that can help the AGC algorithm to
desaturate images more quickly when they are very
over-exposed. However, it uses digital gain to do this which can
confuse our HDR techniques. Therefore make it optional.

Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
Since noise control handling differs between the VC4 and PiSP IPAs,
move the current denoise control handler from ipa base into the vc4 IPA
derived class.

Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
AWB writes this out during prepare, so we may as well read it in AGC
prepare as well. Reading it in process is wrong on the PiSP platform
because process runs before prepare, so the AWB status won't be there
(on vc4 it made no difference).

Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
We need to be able to do things like enable/disable AGC for all the
channels, so most of the AGC controls are updated to be applied to all
channels. There are a couple of exceptions, such as setting explicit
shutter/gain values, which apply only to channel 0.

Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
We add an HdrMode control (to enable and disable HDR processing)
and an HdrChannel, which indicates what kind of HDR frame (short, long
or medium) has just arrived.

Currently the HdrMode supports the following values:

* Off - no HDR processing at all.
* MultiExposureUnmerged - frames at multiple different exposures are
  produced, but not merged together. They are returned "as is".
* MultiExposure - frames at multiple different exposures are merged
  to create HDR images.
* SingleExposure - multiple frames all at the same exposure are
  merged to create HDR images.
* Night - multiple frames will be combined to create "night mode"
  images.

Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
Reviewed-by: Naushir Patuck <naush@raspberrypi.com>
Add support for the following HDR modes in the Raspberry Pi IPA:
- Night mode
- Single exposure mode
- Multi-exposure (merged and unmerged)

The algorithm is updated to expect the HDR short channel to meter
explicitly for highlights. This means that it will not in general
under-expose the short channel more than is actually necessary.

The impact on the HDR algorithm is then that this determines how we
build tonemaps dynamically. The highlights are more-or-less correct
now, so we have to build a power-type curve that gives us the
appropriately configured targets in the lower part of the histogram.

We allow the tuning file to supply the maximum spatial gain value,
rather than the whole curve (though it can supply this if it
wants). Some parameter defaults are tweaked to be generally better
across the range of our cameras.

Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Add the Raspberry Pi 5 ISP (PiSP) pipeline handler to libcamera. To
include this pipeline handler in the build, set the following meson
option:

meson configure -Dpipelines=rpi/pisp

Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Add the Raspberry Pi 5 ISP (PiSP) IPA to libcamera. To include this IPA
in the build, set the following meson option:

meson configure -Dipas=rpi/pisp

Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
The old ctt.py and alsc_only.py scripts are removed.

Instead of ctt.py use ctt_vc4.py or ctt_pisp.py, depending on your
target platform.

Instead of alsc_only.py use alsc_vc4.py or alsc_pisp.py, again
according to your platform.

Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
Reviewed-by: Naushir Patuck <naush@raspberrypi.com>
Added the ability to tune the chromatic aberration correction
within the ctt. There are options for cac_only or to tune as part
of a larger tuning process. CTT will now recognise any files that
begin with "cac" as being chromatic aberration tuning files.

Signed-off-by: Ben Benson <ben.benson@raspberrypi.com>
Reviewed-by: Naushir Patuck <naush@raspberrypi.com>
Changed how users select which platform to tune for. Now users
specify a command line argument, '-t', to specify which target
platform.

Signed-off-by: Ben Benson <ben.benson@raspberrypi.com>
Reviewed-by: Naushir Patuck <naush@raspberrypi.com>
The various boilerplate parts of the tuning file are extended to
include the necessary extra bits for HDR, specifically:

* rpi.denoise has different configurations for HDR modes
* rpi.agc now has extra channels for HDR
* rpi.hdr parameters are added.

Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
Reviewed-by: Naushir Patuck <naush@raspberrypi.com>
The IMX708 sensor driver advertises its module variants (narrow/wide angle lens,
IR block/pass) by modifying the media entity name string. So add duplicate
entries for each variant.

Signed-off-by: Nick Hollinghurst <nick.hollinghurst@raspberrypi.com>
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: Naushir Patuck <naush@raspberrypi.com>
Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Look for the RAW mandatory stream flag in the pipeline handler config
file. If this flag is set, it guarantees that the application will
provide buffers for Unicam Image, so override the minUnicamBuffers and
minTotalUnicamBuffers config parameters in the following way:

- If startup drop frames are required, allocate at least 1 internal buffer.
- If no startup drop frames are required, do not allocate any internal buffers.

Look for the Output 0 mandatory stream flag in in the pipeline handler
config file. If this flag is set, it guarantees that the application
will provide buffers for the ISP, do not allocate any internal buffers
for the device.

Add a new rpi_apps.yaml pipeline handler config file that enables both
these flags.  To use the file, set the following env variable for a
custom build:

export LIBCAMERA_RPI_CONFIG_FILE=/usr/local/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml

or for a packaged install:

export LIBCAMERA_RPI_CONFIG_FILE=/usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml

Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Add an entry for the Omnivision OV64A40 Sensor which has a square pixel
size of 1.008µm.

Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Copy link
Collaborator

@naushir naushir left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks mostly good here apart from a couple of minor things. I've not run it to look at images or anything, but the numbers seem reasonable.

What's the plan to integrate this? Do you want to send it to the mailing list and then we can tag the commits there?

* OV64a40 doesn't output metadata, so we have to use the "unicam parser" which
* works by counting frames.
*/

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment seems to be pulled from another helper, but really it's out of date. Probably best to remove it entirely.

double CamHelperOv64a40::getModeSensitivity(const CameraMode &mode) const
{
/* In binned modes, sensitivity increases by a factor of 2 */
if (mode.binX >= 2 && mode.scaleX >= 4) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment doesn't seem to match what the routine is doing?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll drop both comments ;)

@jmondi
Copy link
Contributor Author

jmondi commented Nov 10, 2023

Looks mostly good here apart from a couple of minor things. I've not run it to look at images or anything, but the numbers seem reasonable.

What's the plan to integrate this? Do you want to send it to the mailing list and then we can tag the commits there?

Yeah I plan to send it to the ml indeed, but only the vc4 part, that's why I went for a pull request here first.

What would your preference be ?

Jacopo Mondi and others added 3 commits November 10, 2023 16:06
Support the OV64A40 sensor with a camera helper to manage the gain
model, light sensitivity, and control delays.

Signed-off-by: Lee Jackson <lee.jackson@arducam.com>
Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Provide the OV64A40 tuning files for the Arducam Omnivision camera
module to operate on the VC4 ISP architecture on Raspberry Pi 4 and
below.

Signed-off-by: Lee Jackson <lee.jackson@arducam.com>
Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Provide the OV64A40 tuning files for the Arducam Omnivision camera
module to operate on the PiSP ISP architecture on Raspberry Pi5.

Signed-off-by: Lee Jackson <lee.jackson@arducam.com>
Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
@naushir
Copy link
Collaborator

naushir commented Nov 10, 2023

I would suggest remove the pisp file, send to the ML to get merged, then once it's merged we rebase on here and include the pisp tuning file.

@jmondi
Copy link
Contributor Author

jmondi commented Nov 10, 2023

I would suggest remove the pisp file, send to the ML to get merged, then once it's merged we rebase on here and include the pisp tuning file.

Will do!

"bayes": 1,
"ct_curve":
[
2300.0, 1.0522, 0.4091,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2300 is an admirably low value, might your incandescent mode want to stretch down this far? (Currently it stops at 2500.)

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At present, the lowest level of our light source is 2300k and the next level is 2700k. We expect it to be able to interpolate to a 2500k light source. If 2300k is deleted, it will start from 2700k. Will there be any negative impact from this?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No - just that you have no way to get a colour temperature lower than 2500. But if you're happy for the AWB searches to stop at 2500, then that's fine and we are happy to merge this as it is!

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No - just that you have no way to get a colour temperature lower than 2500. But if you're happy for the AWB searches to stop at 2500, then that's fine and we are happy to merge this as it is!

Got it, that's acceptable to me.

3000.0, 0.7597, 0.4421,
4000.0, 0.5972, 0.5404,
4150.0, 0.5598, 0.5779,
6500.0, 0.4388, 0.7582
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't a particularly high value, so I wonder if you might want to delete the "cloudy" AWB mode... I can't think it would do anything useful?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, currently our light source can only go up to 6500k, the color temperature range for cloud mode is 7000k to 8000k, I think we can indeed remove that, do you have any suggestions for the calibration process? Especially when there are not many light sources.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it's probably best to remove it. On the other hand, if you've tried it and it actually does something useful, then it's fine to keep it. Getting hold of very blue lamps is tricky, I suppose you could buy some filters but these are often not great either. But I think leaving the top at 6500 is perfectly fine too.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it's probably best to remove it. On the other hand, if you've tried it and it actually does something useful, then it's fine to keep it. Getting hold of very blue lamps is tricky, I suppose you could buy some filters but these are often not great either. But I think leaving the top at 6500 is perfectly fine too.

Got it, thanks for your suggestion.

@jmondi
Copy link
Contributor Author

jmondi commented Nov 14, 2023

I have sent the VC4 part to the mailing removing the "cloudy" AWB mode.

I will send a separate pull request with support for pisp only.

@wallarug
Copy link

Just wondering - what / who are we waiting on for this to be merged in?

I would very much like to use the OV64A40 module cameras I bought 1 month ago.

I am currently stuck with the missing configuration file error. I believe this PR would fix the problem.

image

@kbingham
Copy link
Collaborator

Just wondering - what / who are we waiting on for this to be merged in?

I would very much like to use the OV64A40 module cameras I bought 1 month ago.

I am currently stuck with the missing configuration file error. I believe this PR would fix the problem.

The OV64A40 is merged upstream in libcamera 0.2.0, but Raspberry Pi have not yet had the opportunity to make a release based on top of that yet.

The Pi5 support is expected to be posted to the libcamera development tree very soon, now that the drivers are published, but we're a few weeks away yet from getting that merged still.

So unfortunately at present we have the raspberrypi/libcamera tree here with Pi5 support, and the libcamera mainline tree with OV64A40 support, but not a single tree with both.

As you have purchased this camera from Arducam, you can find an installation method to support this camera documented on their website.

I believe this is where they document it:

Once Raspberry Pi make a new release, this step would not be needed.

@wallarug
Copy link

wallarug commented Feb 10, 2024

Just wondering - what / who are we waiting on for this to be merged in?
I would very much like to use the OV64A40 module cameras I bought 1 month ago.
I am currently stuck with the missing configuration file error. I believe this PR would fix the problem.

The OV64A40 is merged upstream in libcamera 0.2.0, but Raspberry Pi have not yet had the opportunity to make a release based on top of that yet.

The Pi5 support is expected to be posted to the libcamera development tree very soon, now that the drivers are published, but we're a few weeks away yet from getting that merged still.

So unfortunately at present we have the raspberrypi/libcamera tree here with Pi5 support, and the libcamera mainline tree with OV64A40 support, but not a single tree with both.

As you have purchased this camera from Arducam, you can find an installation method to support this camera documented on their website.

I believe this is where they document it:

Once Raspberry Pi make a new release, this step would not be needed.

Thanks Kieran for the reply. Great work you do on libcamera!

So just confirming a few points above:

  1. This PR specifically is not the limiting factor on having the combined Pi 5 + OV64A40 support
  2. ...and will ultimately not be needed / required when bringing over the 0.2.0 stuff anyway.
  3. Waiting on RPi team to merge / rebase on the official libcamera tree before support can be fully supported.
  4. Wait will be "a few weeks away".

Thanks for the link to the Arducam docs. I was sold on the idea and advertising that they 'had upstream support' which is plastered all over their website for the OwlSight (OV) cameras, which they don't have for their Sony sensor version (Hawkeye). Would be good if the Sony 64 MP sensor got added as well, but much less likely given Arducam's ability to support their software.

All good. I will keep an eye out on when the 0.2.0 update comes through on RPi end (rpicam-apps).

@naushir
Copy link
Collaborator

naushir commented Feb 10, 2024

Since Pi 4 support for OV64A40 has been merged upstream, this PR needs a small rework to only add Pi 5 support, plus targeted to the next branch.

We are holding off on an official libcamera release until we have Pi5 support upstream, hopefully this should be very soon now. Otherwise it's quite a large maintenance burden working with releases from a downtream tree that is diverged from upstream.
Our Pi5 support patches sent to upstream (eventually) will include this change I expect, and once that's merged, we can make a new apt release almost immediately adding official support for this camera.

I was sold on the idea and advertising that they 'had upstream support' which is plastered all over their website for the OwlSight (OV) cameras, which they don't have for their Sony sensor version (Hawkeye). Would be good if the Sony 64 MP sensor got added as well, but much less likely given Arducam's ability to support their software.

This raises a good point, perhaps @ArduCAM can elaborate a bit. Pi 5 will have upstream/downstream support for OV64A40, but what about the other Arducam sensors? Is there a plan to add IMX519/Hawkeye? It would be quite unfortunate if their users had to use unofficial versions of libcamera for some of the sensors they sell...

@6by9
Copy link
Collaborator

6by9 commented Feb 10, 2024

I don't believe the Hawkeye (Sony) driver has ever been sent to the linux-media mailing list, therefore doesn't meet the requirements for being merged into libcamera upstream.
There was a debate over whether it could be upstreamed under arducam_64mp or not, but no driver - https://www.spinics.net/lists/linux-media/msg218994.html. Seeing as they appear to have dropped the module, I'd identify it (which is trivial) and rename it to that.

Umang has sent the IMX519 driver to the list a couple of times, and hence the tuning and cam_helper have been merged to libcamera. However it has never been updated for AF support or similar.
The patches are all on https://github.com/Arducam/libcamera, but without a Signed-off-by tag it isn't possible for anyone else to upstream them.

Pivariety modules are a whole issue in themselves. Whilst the Pi kernel does have the driver, support is minimal from our side.

@naushir
Copy link
Collaborator

naushir commented Feb 10, 2024

Umang has sent the IMX519 driver to the list a couple of times, and hence the tuning and cam_helper have been merged to libcamera.

It would be nice if @ArduCAM can upstream any Pi5 tuning file they have, like for the IMX519. Without it, upstream libcamera released will not run for the sensor.

@wallarug
Copy link

Since Pi 4 support for OV64A40 has been merged upstream, this PR needs a small rework to only add Pi 5 support, plus targeted to the next branch.

This is easy enough for me to test. I'll have a play with the RPi 4 tonight and see if it works out of the box without too much messing around. I did note that the RPi Imager now asks what model Pi you are using. Does this mean I need different SD Card images for old and new Pis? (I guess I'll find out).

We are holding off on an official libcamera release until we have Pi5 support upstream, hopefully this should be very soon now. Otherwise it's quite a large maintenance burden working with releases from a downtream tree that is diverged from upstream. Our Pi5 support patches sent to upstream (eventually) will include this change I expect, and once that's merged, we can make a new apt release almost immediately adding official support for this camera.

I tested the v1.3 camera with the RPi 5 and it worked ok out of the box. Will be good to get all the latest changes for other cameras as well.

This raises a good point, perhaps @ArduCAM can elaborate a bit. Pi 5 will have upstream/downstream support for OV64A40, but what about the other Arducam sensors? Is there a plan to add IMX519/Hawkeye? It would be quite unfortunate if their users had to use unofficial versions of libcamera for some of the sensors they sell...

As @6by9 said, the Sony Sensor was never met the requirements, so I kind of accepted this wouldn't work with the default stack. It would be good though. I don't think they have stopped selling it, you can still get them on their website.

@wallarug
Copy link

Since Pi 4 support for OV64A40 has been merged upstream, this PR needs a small rework to only add Pi 5 support, plus targeted to the next branch.

This is easy enough for me to test. I'll have a play with the RPi 4 tonight and see if it works out of the box without too much messing around. I did note that the RPi Imager now asks what model Pi you are using. Does this mean I need different SD Card images for old and new Pis? (I guess I'll find out).

So it doesn't run on RPi 4 either. Same problem with OV64A40 and RPi4 as RPi5. I guess I'll be going down the custom / script method with Arducam for now.

image

@naushir
Copy link
Collaborator

naushir commented Feb 11, 2024

So it doesn't run on RPi 4 either. Same problem with OV64A40 and RPi4 as RPi5

To get this working on a Pi 4 you will need to rebuild libcamera from git.libcamera.org until we have rebased this tree and created release packages.

@wallarug
Copy link

So it doesn't run on RPi 4 either. Same problem with OV64A40 and RPi4 as RPi5

To get this working on a Pi 4 you will need to rebuild libcamera from git.libcamera.org until we have rebased this tree and created release packages.

Hi @naushir, That's what I ended up doing. I have the camera working. Just making it clear that it is NOT upstreamed in RPi4 either.

@wallarug
Copy link

So it doesn't run on RPi 4 either. Same problem with OV64A40 and RPi4 as RPi5

To get this working on a Pi 4 you will need to rebuild libcamera from git.libcamera.org until we have rebased this tree and created release packages.

Hey @naushir - just wondering if there are any updates on progressing this? It's been over 2 weeks since we last messaged.

I am in conversation with Dion at @ArduCAM about this PR as well to see if more support and resources can be thrown at this PR or merging the git.libcamera.org into Raspberry Pi repos going forward.

@naushir
Copy link
Collaborator

naushir commented Feb 26, 2024

This has now been merged in our downstream release of libcamera. You can get this with an apt update.

I'm closing this issue down now.

@naushir naushir closed this Feb 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

9 participants